Computing devices continue to become more ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones, wearable computers (including smartwatches, smart glasses/headsets), global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of interfaces and input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input surface that senses gestures made by a user above the input surface. A further input device is a position detection system which detects the relative position of either touch or non-touch interactions with a non-touch physical or virtual surface. Any of these methods of input can be used generally for the handwritten or hand-drawn input of drawings and text which input is interpreted using a handwriting recognition system or method.
One application of handwriting recognition in computing devices is in the creation of diagrams which are hand-drawn on a computing device to be converted into typeset versions. Diagrams are drawings that explain or show arrangement and relations (as of parts). Diagrams generally include shapes having arbitrary or specific meanings and text with relationships to these shapes. There are many type of diagrams, such as flowcharts, organizational charts, concept maps, spider maps, block/architecture diagrams, mind-maps, block diagrams, Venn diagrams and pyramids. Depictions of some typeset and handwritten examples of possible diagrams are illustrated in FIGS. 1 to 6.
FIGS. 1A and 1B respectively show typeset and handwritten example concept maps 10 variously having shapes, defining diagram blocks or containers 12 and connectors 14, of different type (e.g., straight arrows, curved arrows), which connect or designate relationships between the diagram blocks 12. Further, in FIG. 1B the containers 12 contain text 16. Generally in concept maps the connections between the blocks define conceptually related or dependent elements or themes defined by the text in those blocks. The relationship between blocks may be made precise using labels on connectors. The blocks themselves may not be present in the concept map and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors.
FIGS. 2A and 2B respectively show typeset and handwritten example mind-maps 20 variously having shapes defining diagram blocks or containers 12, connectors 14, of different type (e.g., straight lines, curved lines), which connect or designate relationships between the diagram blocks 12 and paths 18 to certain features or states of the mind maps. Further, in FIG. 2B the containers 12 and paths 18 have associated text 16. Generally in mind-maps the connections between the blocks define possible alternative states or linked ideas from central elements or themes defined by the text in those blocks, and the paths define key features of each alternative state defined by the text on those paths. The blocks themselves may not be present in the mind map and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors and paths.
FIGS. 3A and 3B respectively show typeset and handwritten example flow charts or diagrams 30 variously having shapes, defining diagram blocks or containers 12, of different type (e.g., ovals, rectangles, diamonds), and connectors 14, of different type (e.g., straight arrows, bent arrows, branched lines), which connect or designate relationships between the diagram blocks 12. Further, in FIG. 3B the containers 12 contain text 16; text may also be associated with the connectors. Generally in flow charts the connections between the blocks define procedurally related or dependent elements or steps defined by the text in those blocks. The blocks themselves may not be present in the flow chart and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors.
FIGS. 4A and 4B respectively show typeset and handwritten example organizational charts or tree diagrams 40 variously having shapes, defining diagram blocks or containers 12, and connectors 14, of different type (e.g., straight lines, bent lines, branched lines), which connect or designate relationships between the diagram blocks 12. Further, in FIG. 4B the containers 12 contain text 16. Generally in organizational charts the connections between the blocks define hierarchical relationships of members or functions of an organization or group defined by the text in those blocks. The blocks themselves may not be present in the organizational chart and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors.
FIGS. 5A and 5B respectively show typeset and handwritten example block/architecture drawings 50 variously having shapes, defining diagram blocks or containers 12, having nested relationships (e.g., containers 12 within other containers 12), and connectors 14 which connect or designate relationships between the diagram blocks 12, including between nested blocks. Further, in FIG. 5B the containers 12 and connectors have associated text 16. Generally in architecture drawings the nested blocks define arrangement or possession of device or process components, and the connections between the blocks define functional relationships between the blocks defined by the text in those blocks.
FIGS. 6A and 6B respectively show typeset and handwritten example spider maps 60 variously having shapes, defining diagram blocks or containers 12 and connectors 14 which connect or designate relationships between the diagram blocks 12. Further, in FIG. 6B the containers 12 and connectors have associated text 16. Generally in spider maps the connections between the blocks and/or text define dependent relationships or states from a central element or theme defined by the text.
The diagrams illustrated in FIGS. 1 to 6 are merely examples and other or different elements than those depicted for each diagram type, or different types or forms of the depicted elements themselves, may be present in the diagrams in addition or in the alternative. Further, other definitions of these diagram types are possible as well as combinations thereof. These myriad possible variations of combining the base components of shapes (connections with or without containers) and text in diagrams can cause issues for the accurate recognition of these elements input as hand-drawn or written content to a computing device. Diagrams are particularly used in education and business settings where the user of the computing device creates a diagram, for example, during a lecture or meeting to capture concepts, issues or solutions being discussed. Another frequent use of diagrams is the creation of presentations or reference documentation. Input can be achieved by the user launching a handwritten diagram or sketch application on the computing device which accepts and interprets, either locally in the device or remotely via a communications link of the device, hand-drawn input on a touch sensitive surface or a surface monitored by a relative position detection system.
Conventionally such handwritten diagramming applications are limited in their capabilities to handle the above-described complexity of diagramming and typically constrain users to adopt behaviors or accept compromises which do not reflect the user's original intent. As a result some conventional handwritten diagramming applications force users to navigate menus to select and draw shapes and insert text in relation to shapes. As such, users are unable to draw shapes and connectors naturally or freely. Some other conventional handwritten diagramming applications rely on the order in which users draw different strokes to thereby guide the interpretation for recognition as expected behavior is followed. For example, the user may need to first draw two blocks/boxes before being able to define a connector between those boxes, or may have to draw a box before adding text thereto. This however is difficult for users, as they need to learn and implement the drawing/writing orders required which may need to be re-learnt if the application is not often used, and is non-intuitive, such that the ability to quickly capture diagrams is not supported. For example, a user may wish to prepare presentations on the go with a portable computing device, such as a tablet, or a user may wish to jot down a flow chart that their teacher has drawn in class on a computing device, such as a laptop with a touchscreen, and as such users need to be able to draw clear diagrams with mixed content without being an expert of the dedicated, cumbersome software.
Making the handwritten diagramming application smarter helps support users. That is, the application may be able to distinguish between different shapes, such as between blocks and connectors, and between shapes and text, thereby providing users with more freedom when creating diagrams. For example, U.S. Pat. No. 7,352,902 describes differentiating writing from drawing in input ink by using an ink parser which executes word grouping, writing/drawing classification and drawing grouping. The word grouping is described as being performed by grouping stokes into hierarchies of words, lines and blocks. However, as described in the patent, this grouping of strokes into words during such classification leads to the word groups including non-text strokes, which results in erroneous text recognition when the word groups are sent to a text recognizer, for example. After word grouping, the patent describes that writing/drawing classification is performed. This process is described as including consideration of word, spatial and temporal context features; however these features are mapped to a fuzzy function for classification which means that absolute decisions on whether strokes belong to text or drawings are made at this point, which can lead to misclassification. After classification, the patent describes that drawing grouping is performed by a chart detector to group the drawing stokes into independent objects based on spatial relationships. However, the accuracy of this grouping is influenced by the classification result, such that incorrect objects may be formed.
Even in conventional applications in which hand-drawn shapes and handwritten text are recognized well with reasonable creative freedom offered to users, typically the ability to change the drawn diagrams, such as to edit elements of the diagram to add, omit or replace elements, adapt the diagram to an evolving concept, convert the type of diagram, etc., is limited where only certain operations are available and only available on the typeset version of the diagram, especially with respect to manipulations of the relative positions of diagram elements while retaining recognized relationships, such as connected containers, for example, and not on the handwritten input, so-called digital ink, and/or requires gestures to be learnt or selection to be made via menus, as described above. For example, U.S. Pat. No. 8,014,607 describes an inferred mode protocol which allows certain editing operations to be directly performed on the digital ink. However, the described operations are very limited. Further, no solution is provided for the ability to manipulate the relative position of the diagram elements in digital ink while retaining recognized relationships.
U.S. Pat. No. 7,394,935 describes relative manipulations on the digital ink with respect to resizing and repositioning operations. However, in these operations the digital ink is either merely scaled in accordance with the manipulation, and as such the user would be required to perform further interaction to return the digital ink to its originally drawn dimensions, e.g., moving a container away from its connected container(s) would cause the connector to stretch in both x and y dimensions, or the connectors are ‘reflowed’ by re-computing a backbone (horizontal and vertical lines) that approximates the digital ink of the connector when the connector is resized or changed to a different form (e.g., straight to bent). This requires regeneration of the digital ink, which may be done through normalization of the connector ink through segmentation at high curvature points (cusps) as described in the related U.S. Pat. No. 7,324,691. Accordingly, the resultant manipulated digital ink may be quite different to the originally drawn ink, requiring intervention by users.